Capturing the complexity: Content, type, and amount of instruction and quality of the classroom learning environment synergistically predict third graders’ vocabulary and reading comprehension outcomes

We examined classrooms as complex systems that affect students’ literacy learning through interacting effects of content and amount of time individual students spent in literacy instruction along with the global quality of the classroom-learning environment. We observed 27 third grade classrooms serving 315 target students using two different observation systems. The first assessed instruction at a more micro-level; specifically, the amount of time individual students spent in literacy instruction defined by the type of instruction, role of the teacher, and content. The second assessed the quality of the classroom-learning environment at a more macro level focusing on classroom organization, teacher responsiveness, and support for vocabulary and language. Results revealed that both global quality of the classroom learning environment and time individual students spent in specific types of literacy instruction covering specific content interacted to predict students’ comprehension and vocabulary gains whereas neither system alone did. These findings support a dynamic systems model of how individual children learn in the context of classroom literacy instruction and the classroom-learning environment, which can help to improve observations systems, advance research, elevate teacher evaluation and professional development, and enhance student achievement. NIH Public Access Author Manuscript J Educ Psychol. Author manuscript; available in PMC 2015 August 01. Published in final edited form as: J Educ Psychol. 2014 August ; 106(3): 762–778. doi:10.1037/a0035921. N IH -P A A uhor M anscript N IH -P A A uhor M anscript N IH -P A A uhor M anscript Reading comprehension and vocabulary have been identified as strong predictors of future academic success (NICHD, 2000) as well as of overall school and life outcomes (Beck, McKeown, & Kucan, 2002). Yet, by the end of 4th grade only about 34% of U.S. students are reading and comprehending proficiently (National Center for Education Statistics, 2013). Accumulating research points to the importance of classroom literacy instruction and the opportunities to learn that students receive in the early grades (Connor et al., 2013; NICHD, 2000; Pianta, Belsky, Houts, Morrison, & NICHD-ECCRN, 2007; Snow, 2001; Tuyay, Jennings, & Dixon, 1995). Understanding the classroom learning environment is important and finding ways to elucidate the active ingredients of this environment that predict student outcomes are essential but challenging. In this study, we used a dynamic systems framework (Yoshikawa & Hsueh, 2001), which holds that there are multiple sources of influence on children's learning (Bronfenbrenner & Morris, 2006) including the instruction they receive, how this instruction is delivered (Connor, Piasta, et al., 2009; Reis, McCoach, Little, Muller, & Kaniskan, 2011), the general climate of the classroom (Rimm-Kaufman, Paro, Downer, & Pianta, 2005), teacher characteristics (Raver, Blair, & Li-Grining, 2011), and students themselves (Connor & Morrison, 2012; Justice, Petscher, Schatschneider, & Mashburn, 2011). Further, these sources of influence interact in different ways with some seemingly important factors (e.g., teacher education) having relatively small effects on students’ reading development (Goldhaber & Anthony, 2003) and other factors (e.g., content and minutes of instruction) having large effects (Connor, Morrison, Schatschneider, et al., 2011). High quality literacy instruction should provide students with individualized opportunities to learn that, in turn, influence their reading comprehension and language development (Beck et al., 2002; Beck, Perfetti, & McKeown, 1982; Connor et al., 2013; Snow, 2001). Thus, there is an increasing policy and research focus on how to measure classroom instruction in ways that validly and robustly predict gains in students’ literacy and vocabulary skills (see Crawford, Zucker, Williams, Bhavsar, & Landry, in press; Kane, Staiger, & McCaffrey, 2012; Ramey & Ramey, 2006; Reddy, Fabiano, Dudek, & Hsu, in press; Whitehurst et al., 1988). The aim of this study is to systematically investigate the classroom learning environment as a dynamic system, identify major dimensions of classroom instruction -at both the individual student level and the global classroom level -that may influence students’ literacy achievement and how these dimensions might work together synergistically to support (or fail to support) opportunities for learning that result in gains in third graders’ vocabulary and reading comprehension. Classroom Observation Systems Teacher value-added scores have revealed that there is measurable variability in the effectiveness of teaching, which has direct implications for students’ success or failure (Konstantopoulos & Chung, 2011). However, value-added scores do not reveal what is going on in the classroom and the characteristics of the environment that explain the variability in teachers’ value-added scores. The development of rigorous classroom observation systems that are reliable and have good predictive validity are important because they help to open up the black box of classroom instruction, so to speak, and begin to move us toward what has been described as “shared instructional regimes” (Raudenbush, Connor et al. Page 2 J Educ Psychol. Author manuscript; available in PMC 2015 August 01. N IH -P A A uhor M anscript N IH -P A A uhor M anscript N IH -P A A uhor M anscript 2009). Raudenbush describes historical and recent theories of teaching as “privatized idiosyncratic practice” (p. 172) whereby teachers close their classroom doors and teach in the ways they believe to be best and where the ideal teacher develops his or her own curriculum. The “idiosyncratic” practice of teachers who have a good grasp of the current research, who have expert and specialized knowledge of their content area, and who understand how to use research evidence to inform their practice can be highly effective. However, the privatized idiosyncratic practice of some teachers may be highly ineffective (Piasta, Connor, Fishman, & Morrison, 2009), particularly for children from low SES families whose home learning environment and access to resources is limited and who are more reliant on the instruction they receive at school. Research-based observation tools allow us to illustrate what effective expert practice in the classroom actually looks like so that it can be shared among a community of professionals – both educators and researchers – to improve teaching. There are several well-documented observation systems in use with new systems being developed (Connor, 2013a). These classroom observation systems provide important insights and most of them explain at least modest amounts of the variance in students’ literacy learning. For example, Kane and colleagues (2012) tested several observation instruments, including the Framework for Teaching (FFT; Danielson, 2007), CLASS (Pianta et al., 2008), Protocol for Language Arts Teaching (PLATO, Grossman et al., 2010), Mathematical Quality of Instruction (MQI; Hill, Ball, Bass, & Schilling, 2006), and UTeach Teacher Observation Protocol (UTOP, 2009). Results revealed that although none of the systems designed to assess English/Language Arts instruction correlated with teacher valueadded scores computed using state-mandated assessments of English/Language Arts, they were mildly to moderately positively correlated with teacher value-added scores computed using the SAT-9 reading assessment. Classroom Observations in the Present Study We used two different observation coding systems to test the dynamic systems model of instruction in the current study: the quality of the classroom learning environment (CLE) and ISI/Pathways-observation system (ISI/Pathways, Connor, Morrison, et al., 2009). The first was designed to capture the global quality of the CLE using a rubric that captured elements of the CLE that are generally predictive of student outcomes. The second, ISI/ Pathways, was designed to record the amount of time individual students spent in various types of literacy instruction; the content of this instruction; the role of the teacher; and the context (e.g., whole class, small group) in which instruction was provided. We conjectured, following the dynamic systems model, that classroom opportunities to learn would operate at both student and classroom levels and that the two systems together might better elucidate the complexities of the classroom and effective learning opportunities afforded to students than either system alone. We describe each below. Quality of the Classroom Learning Environment Rating Scale—The CLE rating scale (see Appendix A) was designed to rate the classroom on three dimensions: Teacher Warmth, Responsiveness and Discipline; Classroom Organization; and Teacher Support for Vocabulary and Language Development – with one rating for each scale for the entire Connor et al. Page 3 J Educ Psychol. Author manuscript; available in PMC 2015 August 01. N IH -P A A uhor M anscript N IH -P A A uhor M anscript N IH -P A A uhor M anscript observation of the literacy block. Teacher warmth, responsiveness, and discipline were defined as teachers’ regard for their students, the overall emotional climate of the classroom as well as the way in which they responded to students, particularly with regard to how they responded to student misbehavior and disruptions (Pianta, La Paro, Payne, Cox, & Bradley, 2002). Examples of teacher warmth and responsiveness include being supportive of students, providing positive feedback, clearly communicating what is expected of students, and providing discipline in a positive and supportive way (Rimm-Kaufman et al., 2005). The kinds of discussions and types of questions used, for example, coaching versus telling (Taylor & Pearson, 2002), were measured indirectly through this dimension. Research has shown that students whose teachers were more warm and responsive achieved greater gains in reading skills, including vocabulary, by the end of first grade (Connor, Son, Hindman, & Morrison, 2005). Classroom organization is defined as the degree to which the teacher takes time to give students thorough directions for upcoming activities, has clear rules for behavior, and has established routines that optimize student learning time (Wharton-McDonald, Pressley, & Hampston, 1998). When teachers have strong orienting and organizational skills, they are better able to create an efficient and productive CLE. It has been found that teachers who implement rules and effectively establish routines are less likely to have difficulties with classroom management (Borko & Niles, 1987; Cameron, Connor, Morrison, & Jewkes, 2008). According to Beck and colleagues (2002), teacher support for vocabulary and language development should be “robust,” meaning that instruction should include activities beyond those that encourage rote memorization of words and their definitions and, instead, involve rich contexts that extend beyond the classroom. Such support has the potential to improve language skills overall given that processes where vocabulary knowledge is highly utilized (e.g., during reading comprehension) require skills above and beyond knowing the definitions of words. Therefore, instructional techniques that take into consideration that vocabulary knowledge is part of students’ background knowledge (Stahl, 1999), rather than a singular component, are more likely to be effective. Such techniques encourage students to actively use and think about word meanings and create word associations in multiple contexts. Accumulating evidence further highlights the importance of supporting student language development because vocabulary (and oral language skills in general) are highly predictive of students’ reading comprehension (Biemiller & Boote, 2006; Cain, Oakhill, & Lemmon, 2004), and at the most basic level, allow students to understand grade-level texts. Yet, despite the known importance of vocabulary and language skills to later reading abilities and subsequent academic success, it has been shown that vocabulary instruction is often missing from language arts/literacy classrooms (see Cassidy & Cassidy, 2005/2006; Rupley, Logan, & Nichols, 1998). The Individualizing Student Instruction/Pathways Classroom Observation System (ISI/Pathways)—The ISI/Pathways system (Connor, Morrison, et al., 2009) measures the amount of time (min;sec) individual students within a classroom spend in literacy instruction activities across three dimensions: content of instruction; context; and the role of the teacher and student in the learning activity. The content of instruction dimension Connor et al. Page 4 J Educ Psychol. Author manuscript; available in PMC 2015 August 01. N IH -P A A uhor M anscript N IH -P A A uhor M anscript N IH -P A A uhor M anscript (see Table 1 and Appendix B) captures the specific topic of the literacy instruction that individual students are receiving (e.g., comprehension, vocabulary, text reading). We also coded non-instructional activities, which included off-task or disruptive behaviors, transitions between activities, or time spent when the teacher was giving directions for upcoming activities. The context dimension captures the student grouping arrangement and includes whole class, small group, or individual instruction. Management captures who is controlling the students’ attention during an activity: the teacher and student working together (teacher-child managed), peer-managed (students working with each other), or child-self managed (student managing his/her own attention). These dimensions operate simultaneously (see Table 1) to describe instructional and non-instructional activities observed during reading instruction. For example, the teacher and students discussing a book they just read together would be coded as comprehension (meaning-focused) teacher/ child-managed, whole class activity that lasted for 11 minutes. A key characteristic of ISI/Pathways is that the measurement of instructional time and content is assessed for individual students in the classroom (Connor, Morrison, et al., 2009). Hence, the system is able to capture the learning opportunities afforded to each student; for example, recording that Student A was reading with the teacher while, at the same time, Student B was off task and not redirected. A global classroom-level system would likely capture Student A's instructional opportunities but not Student B's. The more precise measure of each individual student's learning opportunities has been used to identify instructional practices as well as child characteristic X instruction interaction effects on students’ reading achievement (Connor, Morrison, Fishman, et al., 2011; Connor, Morrison, Schatschneider, et al., 2011; Connor, Piasta, et al., 2009). Across studies, amounts, content, context, and types of instruction measured by the ISI/Pathways Observation system predicted student literacy achievement (Connor, Morrison, et al., 2009), particularly the difference between observed and recommended individualized types/content and amounts of instruction (Connor, Piasta, et al., 2009). The closer the observed amount matched the recommended amount, the greater the students’ literacy gains were. We posed the following research question: How does combining measures of the duration of different types of literacy instruction and content for individual students with a more global measure of the CLE synergistically affect students’ reading comprehension and vocabulary outcomes? Using our dynamic systems model of the classroom, we hypothesized that neither the quality of the CLE nor the amount of time individual students spent in different types/content of literacy instruction (ISI/ Pathways) would be strong independent predictors of vocabulary and comprehension outcomes for third grade students. Rather, we hypothesized that there would be interaction effects involving both systems that would significantly and positively predict third graders’ language and comprehension gains. Such interaction effects would, hypothetically, better capture the complexity of classroom instruction and the learning environment. Connor et al. Page 5 J Educ Psychol. Author manuscript; available in PMC 2015 August 01. N IH -P A A uhor M anscript N IH -P A A uhor M anscript N IH -P A A uhor M anscript